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ADAPTIVE INPUT/OUPUT SELECTION 
OF A MULTIMODAL SYSTEM 

5 TECHNICAL FIELD 

This invention relates in general to the field of communications, and more 
particularly to a radio communication device/system that can adaptively select a mode 
of operation. 

BACKGROUND 

Communication devices such as radio communication devices (e.g., cellular 
telephones) are equipped with many input and/or output "modalities" (modes of 
communicating information with the user of the communication device) such as 
speech, audio, video, text messaging and still photography. Some of these radio 
communication devices can also operate in multiple communication systems such as 
Wide Area Local Area Networks (WLANs), 2.5 or 3 rd Generation cellular systems, 
Bluetooth, etc. In such complex devices, selecting the best modality/system for a 
certain situation becomes very important. For example, if a communication device is 
equipped with MPEG-4 (Moving Picture Experts Group version 4), image-based 
rendering and other modalities, selection of the most appropriate modality under 
different system conditions becomes important. 
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Similarly, if the communication device is equipped with voice call, SMS 
(Short Message Service) text messaging and video capability, selecting the best mode 
to use based on what needs to be transmitted and the communication system 
conditions (e.g., noisy channels, not enough bandwidth capability in system to support 
5 video, etc.) also becomes important. Given that changes in system performance are 
affected by noisy environments, changes in bandwidth capabilities, changes in cost of 
service, it becomes very difficult for a user to make all of these decisions alone. Thus, 
a need exists in the art for a communication device that can help address one or more 
of these problems. 

10 



BRIEF DESCRIPTION OF THE DRAWINGS 

The features of the present invention, which are believed to be novel, are set 
forth with particularity in the appended claims. The invention may best be understood 
15 by reference to the following description, taken in conjunction with the accompanying 
drawings, in the several figures of which like reference numerals identify like 
elements, and in which: 

FIG. 1 shows a block diagram of a modality manager handling multiple input 
modalities in accordance with an embodiment of the invention. 
2 0 FIG. 2 shows a block diagram of a modality manager handling multiple output 

modalities in accordance with an embodiment of the invention. 
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FIG. 3 shows a flow diagram highlighting a multimodal client/server 
environment in accordance with an embodiment of the invention. 

FIG. 4 shows a flow diagram showing a messaging interface in accordance 
with an embodiment of the invention. 
5 FIG. 5 shows a flow diagram highlighting a messaging interface for a 

handover event due to poor coverage in accordance with an embodiment of the 
invention. 

FIG. 6 shows a flow chart of an algorithm to determine a network to use in 
accordance with an embodiment of the invention. 
10 FIG. 7 shows a block diagram of an adaptive Multimodal/Multimedia system 

in accordance with an embodiment of the invention. 

FIG. 8 shows a block diagram of a server modality manager communicating 
with a mobile telephone in accordance with an embodiment of the invention. 

FIG. 9 shows a block diagram of a radio communication device in accordance 
15 with an embodiment of the invention. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

While the specification concludes with claims defining the features of the 
invention that are regarded as novel, it is believed that the invention will be better 
2 0 understood from a consideration of the following description in conjunction with the 
drawing figures. 
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Referring to FIG. 1, there is shown a block diagram of a modality management 
system 100 that can be found in an electronic device such as a radio communication 
device. The modality management system 100 includes a Modality Manager (MM) 
1 12 that can handle multiple input modalities 102-1 10. The input modalities shown 
5 include video 102, still picture 104, audio clip 106, voice 108, text 110, etc. The input 
modalities are typically supported by appropriate software and/or hardware found in 
the radio communication device that combines to generate each of the modalities. For 
example, the still picture input modality may be supported by a camera built-into the 
communication device along with the necessary software to convert the image 

1 0 captured into digital information that can be stored and later transmitted by the 
communication device. 

Although specific input modalities have been illustrated, the present invention 
can handle any other type(s) of modalities found in a communication device such as a 
cellular telephone. The MM 112 of the present invention selects from amongst one or 

15 more of the input modalities 102-1 10 based on factors such as the currently available 
communication network(s), available bandwidth and those input modalities that 
provide the lowest cost to the end-user of the radio communication device based on 
the bandwidth, cost and channel conditions. The MM 112 can not only select 
amongst the plurality of input modalities but select from amongst different formats of 

2 0 a particular modality (e.g., MP3, WAV, MPEG-4, JPEG, GIF, etc.). 

In accordance with an embodiment of the invention, MM 112 can 
automatically select one or more of the input modalities according to one or more of 
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the following criteria: bandwidth of the system, bandwidth and service available at the 
moment (e.g., the radio communication device user can be currently out of range of a 
high data rate system and may only be able to transmit at a low data rate, the radio 
channel may be noisy, etc.), cost of the service (i.e., the cost of the service can be 
5 selected by the user or the service provider). Other suitable criteria can be used to 
select input modalities based on a particular design requirement. 

Once an input modality (or modalities) is selected, the MM 1 12 lets the 
application selected by the radio communication device user such as a Multimedia 
Messaging Service (MMS) application 114 or a phone application 116 know what 

10 type of input(s) are acceptable. For example, to create an MMS application 114, the 
application can select the input according to what the MM 112 dictates; so if the MM 
112 accepts only pictures, text and voice, then video and audio clips would not be 
allowed to be transmitted in the MMS application even if the user had selected these 
as part of the MMS application. This could be the case if the MM 112 determined 

1 5 that video could not be transmitted due to lack of bandwidth or a noisy channel 
situation. 

The MM 112 can be implemented as a task (algorithm) operating in a 
microprocessor and/or microcontroller, Digital Signal Processor (DSP) or other 
hardware/software combination. The MM 112 will interface with the input modalities 
2 0 102-1 10 as well as provide control to the end applications 1 14-1 16 selected by the 
communication device user. The MM 112 will also interface with other parts of the 
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communication device (e.g., receiver, microphone) in order to make determinations 
on bandwidth availability, ambient noise issues, etc. 

In FIG. 2, there is shown a multimodal output system 200, in this case the 
Modality Manager 202 is located in a server, for example a server located in a radio 
5 communication system infrastructure (e.g., system controller). A client version MM 
204 resides in the client such as a mobile radio communication device. The client and 
the server side will automatically adapt to the kind of modality that the device can 
handle based on bandwidth, cost and availability. For example, if an MMS is sent to 
the mobile radio communication device, before it is delivered to the mobile radio, the 

10 server MM 202 will communicate with the client MM 204 and find out about the 
system availability with respect to bandwidth and cost of service. 

Once the modality determination has been made by the server MM 202 and . 
client MM 204, the server side MM 202 can deliver the content to the mobile radio, 
adapting the modality transmitted to those approved by MM 202 and MM 204. As an 

15 illustrative example, if an MMS is received at the server MM 202 that has to be 

forwarded to the client MM 204 and includes video, pictures, audio clips and text, 
only those modalities accepted by the client MM 204 will be delivered to the mobile 
radio, the rest can be stored in the server side MM 202 for later accessibility. 

As an illustrative example, if the mobile radio just enters a system which 

2 0 supports high data rate service, the previously stored data that required a high data rate 
service (e.g., video) can be downloaded from the server MM 202 to the mobile radio 
(client MM 204). The mobile radio can request for the rest of the content once the 
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client MM 204 approves it (i.e., mobile radio entering a WLAN coverage area). In 
one embodiment of the invention, the information stored in the server can be 
automatically deleted if after a predetermined period of time the mobile radio cannot 
still accept the information. 

5 

Selection of Appropriate Modality 

When a user of a communication device such as a mobile radio equipped with 
a modality manager launches a communication application such as text messaging, 
etc. (or based on user preferences), the communication application specifies the 

10 bandwidth requirements to the MM located in the mobile radio, based on the existing 
information (e.g., predetermined thresholds stored in the mobile radio and/or 
communication system) or through messages to the mobile radio's modem layer or 
transmitter (depending on the mobile radio design) and obtains several available 
modes (e.g., video, voice, etc.) and cost for the bandwidth. In the case of 

15 unavailability of sufficient bandwidth for the communication application that has been 
requested, it is indicated to the communication application which then can select an 
alternative mode of communication. In one embodiment, the mobile radio user may 
be informed of the unavailability of enough bandwidth via audio and/or visual signals 
such as a blinking Light Emitting Diode (LED) or an audio beep. 

20 In an illustrative example, if a mobile radio is transmitting video and audio 

and the channel bandwidth drops below a predetermined threshold level, then the 
local MM located in the mobile radio upon learning of the bandwidth problem may 
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automatically stop transmitting video and continue to transmit the audio, instead of 
sending video which may not be received with very good quality images. In this 
example, the MM automatically switched modalities in order to maintain the quality 
of the transmitted information at a high level. Since the audio requires less bandwidth 
5 than the video, the mobile radio should be able to continue to transmit audio at a high 
level of quality, even with the bandwidth degradation. A similar automatic modality 
switch can occur due to cost of transmission being greater than a predetermined 
amount using one mode of operation. This illustrative example shows how the 
invention keeps a communication link operational during changes in for example 

10 bandwidth and/or cost of transmission. 

The negotiation between the mobile radio's MM and the system is done until 
the application requirement matches with network bandwidth availability. If there is 
more than one choice available, the pricing information is obtained through the carrier 
database or via user input by the MM in the mobile radio communicating with the 

15 MM in the communication system infrastructure. The available modality choice with 
the lowest cost is then selected. 

In another embodiment of the invention the MM can determine if sensitive 
data is being used by an application and use this information to select which of the 
modalities to use. For example, if the application selected provides a voice or audio 

2 0 output, the MM may restrict those modalities that provide an audio output so others 
may not hear the sensitive information and restrict the modalities available to those 
that can provide protection to the sensitive information such as text messaging. In 
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still another embodiment, the MM can keep track of user preferences for different 
modalities and use this user preference information to select modalities for a particular 
application. 

In FIG. 3, there is shown a flow diagram highlighting a multimodal client 308 
5 and server 306 in the system 300. The client side 308 includes a client MM 302 while 
the server side includes a server MM 304. In this illustrative example, the client can 
comprise a radio communication device such as a mobile radio, while the server side 
306 comprises a communication network controller or other communication system 
infrastructure. A user interface layer 316 which is found in the mobile radio can 

10 provide for events 326 such as launching an application (e.g., text messaging, MMS, 
etc.), terminating an application, changing an application etc. A modem layer 318 
determines such events as signal strength being below a predetermined threshold and 
helps perform handovers between communication networks, etc. An audio processing 
layer 322 determines events 324 such as high ambient noise conditions, discontinuous 

15 transmit (DTX) conditions, etc. The audio processing layer 322 is the task that 

monitors and generates an event if the ambient environment of the mobile radio is too 
noisy. The audio processing layer 322 can be performed by a software routine that 
with the help of a microphone and audio processing circuitry monitors ambient noise 
and is preferably performed as a background task. The noise thresholds are 

2 0 determined and adapted based on the operating conditions of the mobile radio. If the 
noise level is above the predetermined threshold level, an event trigger is sent to the 
client MM 302. The client MM 302 can adapt the User Interface (UI) layer 316 to be 
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text based instead of speech or other audio input that can be affected by the ambient 
noise. Similarly, in the case of video, if the captured video frames are too dark or too 
light with no contrast, the video would be considered noisy and the client MM 302 
would stop sending video and have the user interface layer 316 adjust and only allow 
for one or more other modalities such as voice and/or text. 

The audio processing layer 322, the modem layer 318 and the user interface 
layer 316 all interact with the client MM 302 in order to negotiate modalities, provide 
bandwidth requirements and provide user preferences 310 to the server 306. The 
server MM 304 communicates with the client MM 302 in order to negotiate 
modalities and provide pricing information 312. The audio processing layer 322, the 
modem layer 318 and the user interface layer 316 can all be comprised of software 
and/or hardware that can perform the required tasks. 

Referring now to FIG. 4, there is shown a messaging interface flow diagram 
for a startup event. The different system layers shown include the user interface (UI) 
layer 402, the modem layer 404, the audio processing layer 406, the client MM 408 
and the server MM 410. Upon the electronic device (e.g., mobile radio) startup, the 
radio user may request an MMS application in step 414 as an illustrative example. 
The request for the MMS application is sent to the local client MM 408. The client 
MM 408 sends a message to the modem layer 404 requesting choices for bandwidth 
requirements in step 416. In step 418, the modem layer 404 transmits a message back 
to the client MM 408 that informs it of the available bandwidth options. 



10 



CE11527JDP 



In step 420, the client MM 408 sends a message to the server MM 410 
querying about pricing information for the different mode choices. In response, the 
server MM 410 sends the pricing information for the different mode choices in step 
422 to the client MM 408. In step 424, the client MM 408 determines the lowest cost 
5 option and in step 426 requests a mode change if required to the modem layer 404. In 
step 428, the client MM 408 negotiates media types for the session with the server 
MM 410. Finally, in step 430, the client MM sends the available modality choices to 
the UI layer 402. 

As shown in relation to the message flows in FIG. 4, the architecture results in 
10 an adaptive multimodal system. Events would trigger changes in the state machine. 
Some events that can trigger changes can include handover due to poor coverage in 
the current network, availability of a lower cost network and bandwidth options, low 
signal to noise ratios, user preference triggers such as startup of application and time 
event triggers, and change in bandwidth conditions such as allocation of less 
1 5 bandwidth. 

In FIG. 5, there is shown a flow diagram for a messaging interface for a 
handover event caused by poor reception in the current location of the client MM 508 
which is part of a mobile radio. Like before, the state machine includes the user 
interface layer 502, the modem layer 504, the audio processing layer 506, the client 
2 0 MM 508 and the server MM 510 (located in the server side). In step 514, the modem 
layer 504 informs the client MM 508 of a low signal strength condition. In step 516, 
the client MM 508 requests to the modem layer 504 to obtain information on available 
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networks to handover to. In step 517, the client side layers match the application 
bandwidth requirements with the available bandwidths. In step 518, the client MM 
508 queries the server MM 510 about the cost of current application bandwidth 
requirements for a new mode. In step 520, a low cost option is determined and in step 
522 the client MM 508 informs the modem layer 504 of which network to go to. 
Messages between MM 508 and User Interface layer 502 to match bandwidth 

List below are a couple of the messages (parameter names) used between the 
client MM 508 and the UI layer 502 to match bandwidth requirements: 

1) . MM_UIJBandwith 

Description 

This message sent from Multimodal manager to User Interface 
specifies available bandwidth 

2) . UI_MM_Bandwidth 

- Description 

This message sent from User Interface to Multimodal manager 
specifies the required bandwidth. 

Upon handover between communication networks, "MMJULBandwidth" is 
sent to the application. If the bandwidth fits the application requirements, an 
acknowledgement is received. Otherwise, the application / UI sends a 
"UI_MM_Bandwidth" with its requirements. Depending on the options available, 
the client MM 508 switches to another network. If no options are available for the 
application requested by the user, the application is terminated. The number of 
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messages used will depend on the particular system design requirements and how 
much information needs to be shared between the local and server modality managers, 
etc. 

In FIG. 6 there is shown a flowchart of an illustrative example algorithm to 
determine which communication network to use based on bandwidth requirements 
and cost. In step 602 an event like a request by the user layer to send an MMS 
message is generated. In step 604, the relevancy of the request is made by the client 
MM; the client MM then sends messages to appropriate layers within the 
communication device in step 606. In step 608, the client MM negotiates a new 
session with the application. While in step 610, the client MM sends a message to the 
server MM to determine costs of the different available network(s) that can handle the 
MMS message request. After receiving the cost information, the client MM can then 
automatically transition the mobile radio to the lower cost network in step 612 without 
user intervention. 

Composing an MMS 

Referring to FIG. 7, there is shown a block diagram of a client MM 702 
coupled to a plurality of media inputs 706-714. The media inputs shown include 
video 706, still picture 708, audio clip 710, voice 712, text 714, etc. In this illustrative 
case scenario, if an MMS application 704 is to generate an MMS message, the MM 
702 selects the one or more media inputs 706-714 that will be used for the MMS 
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message. The selection criterion is based on the bandwidth available in the network 
the communication device is presently in, the cost of the service (preferred network) 
and availability. The user of the device will only be allowed to enter those media 
inputs 706-714 which are permitted by the MM 702 after the MM has queried the 
5 systems server MM (not shown). Using this methodology, the electronic device user 
will be able to reduce his airtime costs when the available BW and condition of the 
communication channel is limited. 

Server Modality Manager 

10 Once the MMS message is composed and sent by the electronic device as 

discussed above, the system server will hold the message before sending the MMS to 
the designated receiver. The server modality manager will check with the local 
version of the modality manager of the designated receiving device and verify the type 
of media content the designated receiving device can accept. 

15 Once the server and the receiving unit's MM agree on the content, the server 

can deliver the MMS. If the receiving unit's MM reports the non-availability of a 
channel/service having enough bandwidth to support the message, the server can store 
the MMS for later delivery, deliver the MMS with only that part of the message that 
can be accepted by the receiving device (e.g., text) and then check later for bandwidth 

2 0 availability to deliver the rest of the media content (e.g., video). 

In FIG. 8, there is shown a radio communication system 800 that includes a 
mobile radio (phone) 816 having a local modality manager 804 and multiple media 
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input modalities 806-814. The radio communication system infrastructure 818 
includes a server modality manager 802. The present invention can be used in 
wireless systems like communication system 800 as well as non-wireless systems. 
In FIG. 9 there is shown a block diagram of an electronic device such as a 
5 mobile radio 900 (e.g., cellular telephone) that can take advantage of the modality 
management system of the present invention. Cellular telephone 900 includes an 
antenna 918 which is selectively coupled to a conventional receiver 904 and 
transmitter 906 sections. A controller, such as a microprocessor, microcontroller 
and/or Digital Signal Processor (DSP) 902, provides the overall control for telephone 

10 900. Memory 914 coupled to the controller 902 such as Random Access Memory 
(RAM), Read-Only Memory (ROM), FLASH, etc. stores all of the algorithms and 
variables needed by cellular telephone 900. A display 916 provides visual 
information to the cellular telephone user. 

An audio processing block 908 which can include a vocoder and Analog-to- 

15 Digital (A/D) and Digital-to- Analog (D/A) block provides all the necessary audio 
processing for both incoming and outgoing voice traffic. Coupled to the audio 
processing block 908 is a speaker 912 and microphone 910. The audio processing 
layer 322 may use information gathered by the microphone 910 when making ambient 
noise determinations. 

2 0 Controller 902 performs all of the necessary steps previously described in 

order to provide the adaptive multimodal system previously described. Controller 902 
Memory 914 stores all of the modality information that has been received as well as 
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store predetermined limits regarding acceptable bandwidth and noise levels as well as 
cost information that are used by the MM to make the necessary mode selections. The 
controller 902 will execute all of the necessary algorithms used by the invention in 
accomplishing not only the MM function but the modem layer, user interface layer 
5 and audio processing layers previously described. 

While the preferred embodiments of the invention have been illustrated and 
described, it will be clear that the invention is not so limited. Numerous 
modifications, changes, variations, substitutions and equivalents will occur to those 
skilled in the art without departing from the spirit and scope of the present invention 
10 as defined by the appended claims. 

What is claimed is: 
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